Skip to content

test(verdict): golden-set regression harness for verdict prompt changes - #1135

Draft
kyle-compute wants to merge 2 commits into
stagingfrom
test/verdict-golden-set
Draft

test(verdict): golden-set regression harness for verdict prompt changes#1135
kyle-compute wants to merge 2 commits into
stagingfrom
test/verdict-golden-set

Conversation

@kyle-compute

@kyle-compute kyle-compute commented Aug 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Why this PR exists

What this PR changes

  • Adds a seven-scenario golden set.
  • Covers accept and reject outcomes and high, medium, and low confidence.
  • Pins distinct rules: consistent good trials, repeated task failure, one bad label, all harness errors, a passing nop baseline, weak tests from one trial, and weak tests confirmed by verifier evidence.
  • Adds an offline CI layer that proves every scenario renders into the real prompt and synthetic judge output parses through the production verdict schema.
  • Adds an opt-in live layer that runs the configured verdict model through synthesize_task_verdict, prints every expected/actual result, reports aggregate accept rate, and fails on drift.
  • Supports an alternate prompt file for comparing prompt revisions without editing production code.
  • Disables persistence and cost recording in the live test so it measures judgment behavior without writing S3 or Postgres.

What this PR does not prove

  • Offline tests prove rendering and parsing, not model behavior.
  • Seven hand-written scenarios are not a statistically representative production evaluation.
  • A passing live run does not guarantee identical future output from a nondeterministic provider.
  • This PR changes no production prompt, model, endpoint, schema, persistence, or runtime behavior.

Network, cost, and credentials

  • Normal CI never calls a model.
  • The live suite skips unless the configured OpenAI or Azure OpenAI credentials resolve through the same checks used by production.
  • Running the live suite intentionally makes seven model calls and can incur cost.
  • Each scenario records an error row independently so one provider failure does not hide the rest of the table.

Validation

  • 37 verdict block, fallback, sync, offline-golden, and live-golden tests passed.
  • One live test skipped without credentials, proving the normal-CI gate.
  • The live path was smoke-tested with a throwaway invalid key: all seven cases attempted and drift/errors were reported.
  • The prompt-override seam was verified.

Draft gate

  • Review whether the seven expected labels actually represent team policy.
  • Run the live suite with the production-configured verdict model.
  • Record expected versus actual verdict and confidence for every case.
  • Decide whether the live suite remains manual or becomes a scheduled/approval-gated check.
  • Confirm any accepted model drift updates the fixture and PR explanation together.

Rollback

This PR adds tests only. Revert the test files to remove the harness; no production or stored state needs recovery.

@kyle-compute
test(verdict): add golden-set regression harness for verdict prompt c…
ff2286f 
…hanges

PR #1120 rewrote the verdict prompt and changed TaskVerdictModel from
is_good to verdict=accept|reject with nothing watching whether such a
change shifts the accept/reject line. This adds a golden set of seven
scenarios, each pinning one rule of verdict_prompt.txt to its expected
verdict, consumed by two layers:

- test_verdict_golden_set.py (every CI run, no network): each scenario's
  prompt renders with its defining substance intact, and a synthetic
  judge reply round-trips VerdictBlock's parse path. Guards rendering
  and schema regressions only.
- test_verdict_golden_set_live.py (opt-in): runs the real verdict model
  over the whole set via synthesize_task_verdict, prints a per-case
  table plus the aggregate accept rate, and fails on any drifted
  verdict. Skipped unless the configured OpenAI provider's credentials
  resolve; ODDISH_VERDICT_PROMPT_FILE swaps in an alternative prompt
  for A/B comparison without touching production code.

Claude-Session: https://claude.ai/code/session_01MzJSdrHKEqJrJHU7RRJ758
@vercel

vercel Bot commented Aug 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
oddish-app Ready Ready Preview Aug 10, 2026 9:45pm

cursor[bot]
cursor Bot approved these changes Aug 9, 2026
View reviewed changes

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Risk: low. Approved — Cursor Bugbot passed with no findings requiring human review; Cursor Security Agent was not present. No reviewers assigned.

Open in Web View Automation 

Sent by Cursor Approval Agent: Pull Request Router and Approver

@github-actions

github-actions Bot commented Aug 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Oddish preview

Commit: fad0b1b28de5483483aa59a207bdfd5c23d1cff4

Surface Link Target
Frontend https://pr-1135.oddish.app Vercel preview for fad0b1b
Backend oddish-pr-1135 oddish-pr-1135
Database project vcwffwrhjcrzshgdfcoe project vcwffwrhjcrzshgdfcoe

Vercel deployment URL: https://oddish-7r9i60vxt.oddish.app

Plan:

  • Frontend deploy: true
  • Backend deploy: true
  • Migrations: false

This comment is updated by the PR Preview workflow.

@kyle-compute
kyle-compute marked this pull request as draft August 11, 2026 01:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cursor cursor[bot] cursor[bot] approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@kyle-compute